Techniques for visualizing the operation of neural networks using samples of training data

ABSTRACT

As described, an artificial intelligence (AI) design application exposes various tools to a user for generating, analyzing, evaluating, and describing neural networks. The AI design application includes a network generator that generates and/or updates program code that defines a neural network based on user interactions with a graphical depiction of the network architecture. The AI design application also includes a network analyzer that analyzes the behavior of the neural network at the layer level, neuron level, and weight level in response to test inputs. The AI design application further includes a network evaluator that performs a comprehensive evaluation of the neural network across a range of sample of training data. Finally, the AI design application includes a network descriptor that articulates the behavior of the neural network in natural language and constrains that behavior according to a set of rules.

BACKGROUND Field of the Various Embodiments

Embodiments of the present invention relate generally to computerscience and artificial intelligence and, more specifically, totechniques for creating, analyzing, and modifying neural networks.

Description of the Related Art

In a conventional neural network design process, a designer writesprogram code to develop a neural network architecture that addresses aparticular type of problem. For example, the designer could write Pythoncode to design one or more neural network layers that classify imagesinto different categories. The designer then trains the neural networkusing training data along with target outputs that the neural networkshould produce when processing that training data. For example, thedesigner could train the neural network based on a set of images thatdisplay various landscapes along with labels indicating the types oflandscapes shown in the set of images.

During the training process, a training algorithm updates weightsincluded in the layers of the neural network to improve the accuracywith which the neural network generates outputs that are consistent withthe target outputs. Once training is complete, validation data is usedto determine the accuracy of the neural network. If the neural networkdoes not produce accurate enough results relative to the validationdata, then the neural network can be updated to improve overallaccuracy. For example, the neural network could be trained usingadditional training data until the neural network produces more accurateresults.

Neural networks can have a diverse range of network architectures. A“deep” neural network generally has a complex network architecture thatincludes many different types of layers and an intricate topology ofconnections among the different layers. Some deep neural networks canhave ten or more layers, where each layer can include hundreds orthousands of individual neurons and can be coupled to one or more otherlayers via hundreds or thousands of individual connections. Because deepneural networks can be trained to perform a wide range of tasks with ahigh degree of accuracy, deep neural networks are becoming widelyadopted in the field of artificial intelligence. However, variousproblems arise when designing deep neural networks.

First, the complex network architecture typically associated with deepneural networks can make designing and generating deep neural networksdifficult. When designing a given deep neural network, the designerusually has to write a large volume of complex code that defines howeach layer operates, specifies how the various layers are coupledtogether, and delineates the various operations performed by thedifferent layers. To simplify this process, designers oftentimes rely onone or more programming libraries that expose various tools thatfacilitate deep neural network design. One drawback to using these typesof programming libraries, though, is that the programming librariesgenerally obfuscate the design of a deep neural network from thedesigner and, accordingly, prevent the designer from understanding howthe deep neural network being designed actually operates. Consequently,the designer can have difficulty modifying the deep neural network ifchanges are needed.

Second, the complex neural network architecture typically associatedwith deep neural networks can make the functionality of a given deepneural network difficult to understand. As a result, a typical designercan have trouble analyzing the behavior of a given deep neural networkand determining which components of the deep neural network areresponsible for producing specific behaviors or outcomes. Further,because of the large volume of code normally used to define andimplement a given deep neural network, a typical designer can havedifficulty locating the specific portions of code that are associatedwith any given component of the deep neural network. Thus, when a givendeep neural network does not operate as expected, the designer usuallycannot determine why the deep neural network is not operating asexpected or how to repair or modify the code underlying the deep neuralnetwork.

Third, the complex neural network architecture typically associated withdeep neural networks makes evaluating the performance of a given deepneural network against the training data used when training the deepneural network quite difficult. A conventional training algorithmusually records only the accuracy with which a given deep neural networkgenerates outputs during the training phase. Such conventional trainingalgorithms typically do not provide any additional data to a designer,which limits the ability of the designer to evaluate how well the deepneural network is processing the training data. As a result, mostdesigners cannot determine or explain why a given deep neural networkgenerates a particular output when processing a given sample of trainingdata.

Fourth, the complex neural network architecture typically associatedwith given deep neural networks can be difficult for a designer tocharacterize and describe. Consequently, a typical designer can havetrouble explaining to others how a given deep neural network operates.For the reasons discussed above, the designer oftentimes does notunderstand how the deep neural network operates and, therefore, cannotfully articulate or explain the various functional characteristics ofthe deep neural network.

As the foregoing illustrates, what is needed in the art are moreeffective techniques for generating, analyzing, and modifying neuralnetworks.

SUMMARY

Various embodiments include a computer-implemented method for analyzinghow a neural network has been trained, including causing the neuralnetwork to execute an inference operation based on a plurality ofsamples of training data to generate a plurality of portions ofactivation data, wherein each sample of training data corresponds to adifferent portion of activation data, generating a position value foreach sample of training data based on the portion of activation datacorresponding to the sample of training data, and generating a graphicaluser interface that displays each sample of training data positionedwithin the graphical user interface based on the position valuegenerated for the sample of training data.

At least one technological advantage of the disclosed techniquesrelative to the prior art is that the disclosed AI design applicationperforms detailed analyses of how a given neural network operates duringthe training phase, thereby enabling a designer to better understand whythe neural network generates specific outputs based on particularinputs.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the variousembodiments can be understood in detail, a more particular descriptionof the inventive concepts, briefly summarized above, may be had byreference to various embodiments, some of which are illustrated in theappended drawings. It is to be noted, however, that the appendeddrawings illustrate only typical embodiments of the inventive conceptsand are therefore not to be considered limiting of scope in any way, andthat there are other equally effective embodiments.

FIG. 1 illustrates a system configured to implement one or more aspectsof the various embodiments;

FIG. 2 is a more detailed illustration of the AI design application ofFIG. 1 , according to various embodiments;

FIG. 3 is a more detailed illustration of the network generator of FIG.2 , according to various embodiments;

FIG. 4 is a screenshot illustrating how the network generation GUI ofFIG. 2 facilitates the generation of a neural network, according tovarious embodiments;

FIG. 5 is a screenshot illustrating how the network generation GUI ofFIG. 2 facilitates the generation of an AI model, according to variousother embodiments;

FIG. 6 is a screenshot of various underlying data associated with one ofthe agents of FIG. 5 , according to various embodiments;

FIG. 7 is flow diagram of method steps for generating and modifying aneural network via a graphical user interface, according to variousembodiments;

FIG. 8 is a more detailed illustration of the network analyzer of FIG. 2, according to various embodiments;

FIG. 9 is a screenshot illustrating how the network analysis GUI of FIG.2 facilitates inspection of a neural network, according to variousembodiments;

FIG. 10 is a screenshot illustrating how the network analysis GUI ofFIG. 2 exposes the underlying functionality of an agent, according tovarious embodiments;

FIG. 11 is a screenshot illustrating how the network analysis GUI ofFIG. 2 exposes a set of agents for processing test inputs, according tovarious embodiments;

FIG. 12 is a screenshot illustrating how the network analysis GUI ofFIG. 2 applies an agent to process a test input, according to variousembodiments;

FIG. 13 is a screenshot illustrating how the network analysis GUI ofFIG. 2 applies another agent to process a test input, according tovarious embodiments;

FIG. 14 is a screenshot illustrating how the network analysis GUI ofFIG. 2 applies a different agent to a test input, according to variousother embodiments;

FIGS. 15A-15B set forth a flow diagram of method steps for analyzing aneural network via a graphical user interface, according to variousembodiments;

FIG. 16 is a more detailed illustration of the network evaluator of FIG.2 , according to various embodiments;

FIG. 17 is a screenshot illustrating how network evaluation GUI of FIG.2 facilitates exploration of training data, according to variousembodiments;

FIG. 18 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 receives input via a sample map, according to variousembodiments;

FIG. 19 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data assigned a high confidencevalue, according to various embodiments;

FIG. 20 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data assigned a low confidencevalue, according to various embodiments;

FIG. 21 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data labeled overconfident,according to various embodiments;

FIG. 22 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 indicates samples of training data that promote a selected neuralnetwork output, according to various embodiments;

FIG. 23 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data sorted based on a neuralnetwork output, according to various embodiments;

FIG. 24 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 indicates samples of training data that meet specific activationcriteria, according to various embodiments;

FIG. 25 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data sorted based on an expression,according to various embodiments;

FIG. 26 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays relevant portions of a training sample, according tovarious embodiments;

FIGS. 27A-27B set forth a flow diagram of method steps for evaluating aneural network relative to a set of training data via a graphical userinterface, according to various embodiments;

FIG. 28 is a more detailed illustration of the network descriptor ofFIG. 2 , according to various embodiments;

FIG. 29 is a screenshot illustrating how the network description GUI ofFIG. 2 facilitates the constraining of neural network behavior undervarious circumstances, according to various embodiments;

FIG. 30 is a screenshot illustrating how the network description GUI ofFIG. 2 articulates neural network behavior, according to variousembodiments;

FIG. 31 is a screenshot illustrating how the network description GUI ofFIG. 2 represents a derived fact, according to various embodiments;

FIG. 32 is a screenshot illustrating how the network description GUI ofFIG. 2 depicts performance data associated with the training of a neuralnetwork, according to various embodiments;

FIG. 33 is a screenshot illustrating how the network description GUI ofFIG. 2 depicts other performance data associated with the training of aneural network, according to various other embodiments;

FIG. 34 is a screenshot illustrating how the network description GUI ofFIG. 2 displays the amount of memory consumed when executing a neuralnetwork, according to various embodiments;

FIG. 35 is a screenshot illustrating how the network description GUI ofFIG. 2 represents different versions of a given neural network,according to various embodiments;

FIG. 36 is a screenshot illustrating how the network description GUI ofFIG. 2 displays comparative performance data associated with differentversions of a given neural network, according to various embodiments;

FIG. 37 is a screenshot illustrating how the network description GUI ofFIG. 2 displays other comparative performance data associated withdifferent versions of a given neural network, according to various otherembodiments; and

FIGS. 38A-38B set forth a flow diagram of method steps for articulatingand constraining the behavior of a neural network via a graphical userinterface, according to various embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the various embodiments.However, it will be apparent to one skilled in the art that theinventive concepts may be practiced without one or more of thesespecific details.

As noted above, deep neural networks can have complex networkarchitectures that include numerous layers and intricate connectiontopologies. Consequently, a deep neural network can be difficult for adesigner to generate. Further, once the deep neural network isgenerated, the complexity of the network architecture associated withthe deep neural network can be difficult for the designer to analyze andunderstand. With a limited ability to analyze and understand the deepneural network, the designer can have further difficulty evaluating howwell the deep neural network performs an intended task. Finally, lackingan explicit understanding of how the deep neural network operates, thedesigner cannot easily characterize the operation of the deep neuralnetwork or describe that operation to others.

To address these issues, various embodiments include an artificialintelligence (AI) design application that exposes various tools to auser for generating, analyzing, evaluating, and describing neuralnetworks. The AI design application includes a network generator thatgenerates and/or updates program code that defines a neural networkbased on user interactions with a graphical depiction of the networkarchitecture. The AI design application also includes a network analyzerthat analyzes the behavior of the neural network at the layer level,neuron level, and weight level in response to test inputs. The AI designapplication further includes a network evaluator that performs acomprehensive evaluation of the neural network across a range of sampleof training data. Finally, the AI design application includes a networkdescriptor that articulates the behavior of the neural network innatural language and constrains that behavior according to a set ofrules.

At least one technological advantage of the disclosed techniquesrelative to the prior art is that the disclosed AI design applicationcan generate complex neural network architectures without requiring adesigner user to write or interact with large amounts of program code.Another technological advantage of the disclosed techniques relative tothe prior art is that the disclosed AI design application provides adesigner with detailed information about the underlying operations andfunctions of the individual components of a given neural networkarchitecture. Accordingly, the AI design application enables a designerto develop and better understanding of how the neural network operates.Another technological advantage of the disclosed techniques relative tothe prior art is that the disclosed AI design application performsdetailed analyses of how a given neural network operates during thetraining phase, thereby enabling a designer to better understand why theneural network generates specific outputs based on particular inputs.Yet another technological advantage of the disclosed techniques relativeto the prior art is that the disclosed AI design applicationautomatically generates natural language descriptions characterizing howa given neural network operates and functions. Among other things, thesedescriptions help explain the operations of the neural network to adesigner and enable the designer to articulate and explain thefunctional characteristics of the neural network to others. Thesetechnological advantages represent one or more technologicaladvancements over prior art approaches.

System Overview

FIG. 1 illustrates a system configured to implement one or more aspectsof the various embodiments. As shown, a system 100 includes a client 110and a server 130 coupled together via a network 150. Client 110 orserver 130 may be any technically feasible type of computer system,including a desktop computer, a laptop computer, a mobile device, avirtualized instance of a computing device, a distributed and/orcloud-based computer system, and so forth. Network 150 may be anytechnically feasible set of interconnected communication links,including a local area network (LAN), wide area network (WAN), the WorldWide Web, or the Internet, among others. Client 110 and server 130 areconfigured to communicate via network 150.

As further shown, client 110 includes a processor 112, input/output(I/O) devices 114, and a memory 116, coupled together. Processor 112includes any technically feasible set of hardware units configured toprocess data and execute software applications. For example, processor112 could include one or more central processing units (CPUs), one ormore graphics processing units (GPUs), and/or one or more parallelprocessing units (PPUs). I/O devices 114 include any technicallyfeasible set of devices configured to perform input and/or outputoperations, including, for example, a display device, a keyboard, and atouchscreen, among others.

Memory 116 includes any technically feasible storage media configured tostore data and software applications, such as, for example, a hard disk,a random-access memory (RAM) module, and a read-only memory (ROM).Memory 116 includes a database 118(0), an artificial intelligence (AI)design application 120(0), an AI model 122(0), and a graphical userinterface (GUI) 124(0). Database 118(0) is a file system and/or datastorage application that stores various types of data. AI designapplication 120(0) is a software application that, when executed byprocessor 112, interoperates with a corresponding software applicationexecuting on server 130 to generate, analyze, evaluate, and describe oneor more AI models. AI model 122(0) includes one or more artificialneural networks configured to perform general-purpose or specializedartificial intelligence-oriented operations. GUI 124(0) allows a user tointerface with AI design application 120(0).

Server 130 includes a processor 132, I/O devices 134, and a memory 136,coupled together. Processor 132 includes any technically feasible set ofhardware units configured to process data and execute softwareapplications, such as one or more CPUs, one or more GPUs, and/or one ormore PPUs. I/O devices 134 include any technically feasible set ofdevices configured to perform input and/or output operations, such as adisplay device, a keyboard, or a touchscreen, among others.

Memory 136 includes any technically feasible storage media configured tostore data and software applications, such as, for example, a hard disk,a RAM module, and a ROM. Memory 136 includes a database 118(1), an AIdesign application 120(1), an AI model 122(1), and a GUI 124(1).Database 118(1) is a file system and/or data storage application thatstores various types of data, similar to database 118(1). AI designapplication 120(1) is a software application that, when executed byprocessor 132, interoperates with AI design application 120(0) togenerate, analyze, evaluate, and describe one or more AI models. AImodel 122(1) includes one or more artificial neural networks configuredto perform general-purpose or specialized artificialintelligence-oriented operations. GUI 124(1) allows a user to interfacewith AI design application 120(1).

As a general matter, database 118(0) and 118(1) represent separateportions of a distributed storage entity. Thus, for simplicity,databases 118(0) and 118(1) are collectively referred to herein asdatabase 118. Similarly, AI design applications 120(0) and 120(1)represent separate portions of a distributed software entity that isconfigured to perform any and all of the inventive operations describedherein. As such, AI design applications 120(0) and 120(1) arecollectively referred to hereinafter as AI design application 120. AImodels 122(0) and 122(1) likewise represent a distributed AI model thatincludes one or more neural networks. Accordingly, AI models 122(0) and122(1) are collectively referred to herein as AI model 122. GUIs 124(0)and 124(1) similarly represent distributed portions of one or more GUIs.GUIs 124(0) and 124(1) are collectively referred to herein as GUI 124.

In operation, AI design application 120 generates AI model 122 based onuser input that is received via GUI 124. GUI 124 exposes design andanalysis tools that allow the user to create and edit AI model 122,explore the functionality of AI model 122, evaluate AI model 122relative to training data, and generate various data describing and/orconstraining the performance and/or operation of AI model 122, amongother operations. Various modules within AI design application 120 thatperform the above operations are described in greater detail below inconjunction with FIG. 2 .

FIG. 2 is a more detailed illustration of the AI design application ofFIG. 1 , according to various embodiments. As shown, AI designapplication 120 includes a network generator 200, a network analyzer210, a network evaluator 230, and a network descriptor 230. As alsoshown, AI model 122 includes one or more agents 240, and GUI 124includes network generation GUI 202, network analysis GUI 212, networkevaluation GUI 222, and network description GUI 232.

In operation, network generator 200 renders network generation GUI 202to provide the user with tools for designing and connecting agents 240within AI model 122. A given agent 240 may include a neural network 242that performs various AI-oriented tasks. A given agent 240 may alsoinclude other types of functional elements that perform generic tasks.Network generator 200 trains neural networks 242 included in specificagents 240 based on training data 250. Training data 250 can include anytechnically feasible type of data for training neural networks. Forexample, training data 250 could include the Modified National Instituteof Standards and Technology (MNIST) digits training set. Networkgenerator 200 and network generation GUI 202 are described in greaterdetail below in conjunction with FIGS. 3-7 .

When training is complete, network analyzer 210 renders network analysisGUI 212 to provide the user with tools for analyzing and understandinghow a neural network within a given agent 240 operates. In particular,network analyzer 210 causes network analysis GUI 212 to display variousconnections and weights within a given neural network 242 and tosimulate the response of the given neural network 242 to various inputs,among other operations. Network analyzer 210 and network analysis GUI212 are described in greater detail below in conjunction with FIGS.8-15B.

In addition, network evaluator 220 renders network evaluation GUI 222 toprovide the user with tools for evaluating a given neural network 242relative to training data 250. More specifically, network evaluator 220receives user input via network evaluation GUI 222 indicating aparticular portion of training data 250. Network evaluator 220 thensimulates how the given neural network 242 responds to that portion oftraining data 250. Network evaluator 220 can also cause networkevaluation GUI 222 to filter specific portions of training data 250 thatcause the given neural network 242 to generate certain types of outputs.Network evaluator 220 and network evaluation GUI 222 are described ingreater detail below in conjunction with FIGS. 16-27B.

In conjunction with the above, network descriptor 230 analyzes a givenneural network 242 associated with an agent 240 and generates a naturallanguage expression that describes the performance of the neural network242 to the user. Network descriptor 230 can also provide various “commonsense” facts to the user related to how the neural network 242interprets training data 250. Network descriptor 230 outputs this datato the user via network description GUI 232. In addition, networkdescriptor 230 can obtain rule-based expressions from the user vianetwork description GUI 232 and then constrain network behavior based onthese expressions. Further, network descriptor 230 can generate metricsthat quantify various aspects of network performance and then displaythese metrics to the user via network description GUI 232. Networkdescriptor 230 and network descriptor GUI 232 are described in greaterdetail below in conjunction with FIGS. 38-38B.

Referring generally to FIGS. 1-2 , AI design application 120advantageously provides the user with various tools for generating,analyzing, evaluating, and describing neural network behavior. Thedisclosed techniques differ from conventional approaches to generatingneural networks which generally obfuscate network training andsubsequent operation from the user.

Generating and Modifying Neural Networks

FIGS. 3-7 set forth various techniques implemented by network generator200 of FIG. 2 when generating a neural network 242 based on trainingdata 250 and subsequently modifying that neural network. As described ingreater detail herein, network generator 200 generates networkgeneration GUI 202 in order to facilitate the generation andmodification of the neural network.

FIG. 3 is a more detailed illustration of the network generator of FIG.1 , according to various embodiments. As shown, network generator 200includes a compiler engine 300, a synthesis engine 310, a trainingengine 320, and a visualization engine 330.

In operation, visualization engine 330 generates network generation GUI202 and obtains agent definitions 340 from the user via networkgeneration GUI 202. Compiler engine 200 compiles program code includedin a given agent definition 340 to generate compiled code 302. Compilerengine 200 is configured to parse, compile, and/or interpret anytechnically feasible programming language, including C, C++, Python andassociated frameworks, JavaScript and associated frameworks, and soforth. Synthesis engine 310 generates initial network 312 based oncompiled code 302 and on or more parameters that influence how that codeexecutes. Initial network 312 is untrained and may not perform one ormore intended operations with a high degree of accuracy.

Training engine 330 trains initial network 312 based on training data250 to generate trained network 322. Trained network 322 may perform theone or more intended operations with a higher degree of accuracy thaninitial network 312. Training engine 330 may perform any technicallyfeasible type of training operation, including backpropagation, gradientdescent, and so forth. Visualization engine 330 updates networkgeneration GUI 202 in conjunction with the above operations tographically depict the network architecture defined via agent definition340 as well as to illustrate various performance attributes of trainednetwork 322. FIGS. 4-6 set forth various exemplary screenshots ofnetwork generation GUI 202.

FIG. 4 is a screenshot illustrating how the network generation GUI ofFIG. 2 facilitates the generation of a neural network, according tovarious embodiments. As shown, a GUI panel 400 includes model definitionpanel 410, hyperparameter panel 420, and description panel 430. GUIpanel 400 resides within network generation GUI 202.

Model definition panel 410 is a text field that obtains a description ofthe network architecture from the user. For example, model definitionpanel 410 could receive program code that defines one or more layersassociated with a neural network and how those layers are coupledtogether. Alternatively, model definition panel 410 could receivemathematical notation that mathematically described the neural networkarchitecture. In one embodiment, model definition panel 410 exposes aportion of a network and omits other portions that do not need to beedited by the user, such as module imports, among others. Hyperparameterpanel 420 is a text field that receives various hyperparameters thatinfluence how the neural network is trained. For example, hyperparameterpanel 420 could receive a number of training epochs and/or a learningrate from the user. Description panel 430 includes a natural languagedescription of the neural network that is automatically produced bynetwork generator 200 based, at least in part, on the contents of modeldefinition panel 410.

Network generator 200 implements the technique described above inconjunction with FIG. 3 to generate a neural network, such as thoseshown in FIG. 2 , based on the description of the network architectureobtained from the user. Network generator 200 also updates GUI panel 400to include network architecture 440. Network architecture 440graphically depicts the type and arrangement of layers in the neuralnetwork and any other topological information associated with the neuralnetwork. In the example shown, network architecture 440 includes aninput layer 442, two convolution layers 442 and 446, a max pooling layer448, a dropout layer 450, and an activation layer 452.

Network generator 300 is configured to dynamically modify the underlyingneural network 242 defined in model definition panel 410 based on userinteractions with network architecture 440. For example, networkgenerator 300 could receive user input indicating that a particularportion of network architecture 440 should be modified. In response,network generator 300 modifies the underlying neural network and alsoupdates the definition included in model definition panel 410 in acorresponding fashion. In addition, network generator 300 is configuredto dynamically modify network architecture 440 based on userinteractions with model definition panel 410. For example, GUI panel 400could receive input indicating one or more changes to the descriptionset forth in model definition panel 410. In response, network generator300 modifies the underlying neural network and also updates networkarchitecture 440 to reflect the changes.

Network generator 300 can implement the above techniques via networkgeneration GUI 302 in order to create and modify neural networks 242included in agents 240. Network generator 300 can also define othertypes of agents that perform generic operations, as previouslymentioned. Via network generation GUI 302, network generator 300 obtainsa configuration of agents 240 that implements a particular AI model 122,as described in greater detail below in conjunction with FIG. 5 .

FIG. 5 is a screenshot illustrating how the network generation GUI ofFIG. 2 facilitates the generation of an AI model, according to variousother embodiments. As shown, a GUI panel 500 includes agent panel 510,design area 520, and training data panel 530. GUI panel 500 is includedin network generation GUI 202. The AI model discussed in conjunctionwith this example performs various operations related to determininglicense plate information based on photographs of automobiles.

Agent panel 510 includes a list of available agents 240 that performspecific tasks, including agent 240(0) (“find cars”), agent 240(1)(“find license plates”), agent 240(2) (“read license plates”), and agent240(3) (“look up registration”). Agents 240(0) through 240(2) aregenerally neural-network based agents that perform image processing andtagging operations. Agent 240(3), by contrast, includes program codethat interfaces with an external server to obtain registrationinformation associated with a given license plate.

Based on user interactions with network generation GUI 202, networkgenerator 200 arranges various agents 240 selected from agent panel 510to produce AI model 122 within a design area 520. In this example, AImodel 122 is a collection of neural networks 242 and other functionalunits that, once trained, can analyze photographs of automobiles toextract license plate numbers and then obtain registration informationassociated with those license plate numbers. In operation, agent 240(0)locates cars or other automobiles within input images. Agent 240(1)locates license plates associated with those cars and other automobiles.Agent 240(2) extracts text from the located license plates. Agent 240(3)queries a server to obtain registration information for the extractedlicense plate numbers.

Network generator 200 trains the neural network-based agents 240 withinAI model 122(0) based on training data 250. Exemplary training data isdisplayed within training data panel 530. As is shown, training datapanel 530 depicts various sample photographs of automobiles. In oneembodiment, the license plate of each automobile may be labeled tofacilitate the training process.

Network generator 200 can expose underlying data associated with any ofagents 240 in response to user input. For example, in response to a userselection of agent 240(3), network generator 200 could display programcode that queries a server to obtain registration information in themanner discussed previously. Network generator 200 could receivemodifications to that program code and then update AI model 122accordingly. In response to a user selection of a neural network-basedagent 240, network generator 200 exposes underlying data associated withthat agent, including the underlying neural network 242, via various GUIelements described below in conjunction with FIG. 6 .

FIG. 6 is a screenshot of various underlying data associated with one ofthe agents of FIG. 5 , according to various embodiments. As shown, GUIpanel 500 of FIG. 5 includes a window 600 that is superimposed overother GUI elements of GUI panel 500. Window 600 includes variousunderlying data associated with the selected agent. In the exampleshown, agent 250(2) is selected (“read license plates”).

Window 600 includes a model definition panel 610 that includes programcode defining agent 250(2), hyperparameters panel 620 that definesvarious hyperparameters used when training the associated neuralnetwork, and description panel 630 that describes various attributes ofthat neural network. Window 600 also includes network architecture 640.In like fashion as described above in conjunction with FIG. 4 , networkgenerator 200 can update the model definition set forth in modeldefinition window 610 based on user interactions with networkarchitecture 640. For example, in response to user input indicating thata layer of network architecture 640 should be removed, network generator200 could delete a corresponding portion of the model definition.

Referring generally to FIGS. 3-6 , the above techniques provide the userwith a convenient mechanism for generating and updating neural networksthat are integrated into potentially complex AI models 122 that includenumerous agents 240. Further, these techniques allow the user to modifyprogram code that defines a given agent 240 via straightforwardinteractions with a graphical depiction of the corresponding networkarchitecture. Network generator 200 performs the various operationsdescribed above based on user interactions conducted via networkgeneration GUI 202. The disclosed techniques provide the user withconvenient tools for designing and interacting with neural networks thatexpose network information to the user rather than allowing thatinformation to remain hidden, as generally found with prior arttechniques. The operation of network generator 200 is described ingreater detail below in conjunction with FIG. 7 .

FIG. 7 is flow diagram of method steps for generating and modifying aneural network via a graphical user interface, according to variousembodiments. Although the method steps are described in conjunction withthe systems of FIGS. 1-6 , persons skilled in the art will understandthat any system configured to perform the method steps in any orderfalls within the scope of the present embodiments.

As shown, a method 700 begins at step 702, where design generator 200 ofFIG. 3 generates design generation GUI 202 to depict a set of agents anda set of training data. A given agent may include a neural network thatperforms neural network-oriented operations or program code that, whenexecuted, performs any technically feasible operation. Design generationGUI 202 also includes a design area where agents can be arranged andcoupled together to generate an AI model 122.

At step 704, network generator 200 receives a configuration of agents240 forming an AI model via network generation GUI 202. When coupledtogether, the output of a given agent can be provided as the input toanother agent, thereby forming a pipeline of processing stages. In oneembodiment, design generation GUI 202 may allow the user to drag anddrop agents to different locations within the design area and dragconnections between outputs and inputs of agents.

At step 706, design generator 200 receives an agent definition via userinteraction with design generation GUI 202. The agent definitiongenerally includes program code that, when executed, performs one ormore operations associated with the overarching operation of the AImodel. The agent definition discussed herein defines a neural network242 that needs to be trained based on training data. In some cases,agent definitions can define specific functions that perform a givenoperation when executed, as discussed.

At step 708, network generator 200 compiles the agent definitionreceived at step 706 to generate compiled code. The compiled codeimplements the various layers of a neural network 242 and variousconnections between those layers. The compiled code generally targetsunderlying hardware associated with a particular computer system wherethe AI model executes.

At step 710, network generator 200 synthesizes the compiled code togenerate an initial version of the network. In so doing, networkgenerator 200 executes the compiled code with one or more inputparameters, including configuration parameters as well as trainingparameters, to instantiate an instance of the network. The initialversion of the network is untrained and may not perform inferenceoperations accurately until after training is complete.

At step 712, network generator 200 trains the initial version of thenetwork based on training data to generate a trained version of network.The training data generally includes samples of data for the network toprocess and potentially includes labels indicating correct outputs thatthe network should produce. Network generator 200 can train the networkusing backpropagation, gradient descent, or any other technicallyfeasible approach to training.

At step 714, network generator 200 updates design generation GUI 202 toexpose underlying data associated with a user-selected agent 240. Forexample, design generator 200 could generate a window that includes amodel definition panel and a hyperparameter panel, among others, viawhich the user can modify the neural network 242 associated with theagent 240. The window could further include a graphical depiction of thenetwork architecture with which the user can interact to applymodifications to the neural network. This particular example isdescribed above in conjunction with FIG. 6 .

At step 716, network generator 200 receives a modification to thenetwork architecture via a user interaction with design generation GUI202. For example, the user could select a layer of the networkarchitecture depicted in network generation GUI 202 and then remove thatlayer from the network architecture. In another example, the user couldselect a portion of the network architecture and then modify one or moreparameters associated with that portion of the network architecture.

At step 718, network generator 200 updates and re-compiles the agentdefinition based on the modification to the network architecturereceived at step 716. For example, if the user removes a layer of thenetwork architecture via interaction with design generation GUI 202,then network generator 200 could update the agent definition to removeone or more corresponding lines of code that define that layer.

As a general matter, the techniques described above for generating andmodifying neural networks allow users to design and modify neuralnetworks much faster than conventional approaches permit. Among otherthings, network generator 200 provides simple and intuitive tools forperforming complex tasks associated with network generation.Additionally, network generator 200 conveniently allows modificationsthat have been made to a neural network architecture to be seamlesslypropagated back to a corresponding agent definition. Once the network istrained in the manner described, network analyzer 210 performs varioustechniques for analyzing network functionality, as described in greaterdetail below in conjunction with FIGS. 8-15B.

Inspecting and Analyzing Components of Neural Networks

FIGS. 8-15B set forth various techniques implemented by network analyzer210 of FIG. 2 when analyzing a neural network that is trained based ontraining data 250. As described in greater detail herein, networkanalyzer 210 generates network analysis GUI 212 in order to facilitatethe analysis and inspection of the neural network.

FIG. 8 is a more detailed illustration of the network analyzer of FIG. 2, according to various embodiments. As shown, network analysis engine210 includes an inference engine 800, an approximation engine 810, alanguage engine 820, and a visualization engine 830.

In operation, inference engine 800 generates activation data 802 byperforming an inference operation with an agent 240 and test input 804.In particular, inference engine provides test input 804 to a neuralnetwork 242 associated with agent 240 and then determines the responseof that neural network to test input 804. Activation data 802 indicatesa probability distribution of responses associated with a particularlayer of the neural network. Inference engine 800 transmits activationdata 802 to visualization engine 830 for subsequent incorporation intonetwork analysis GUI 212. Inference engine 800 also transmits activationdata 802 to approximation engine 810 and language engine 820.

Approximation engine 810 analyzes activation data 802 in conjunctionwith training data 250 to generate training samples 812. Trainingsamples 812 include a subset of training data 250 that cause neuralnetwork 242 to generate activation data that is substantially similar toactivation data 802. A given activation data may be considered“substantially similar” to activation data 802 when a numericaldifference between the given activation data and activation data 802 isless than a threshold difference value. In one embodiment, training data250 may include activation levels associated with each sample previouslyrecorded during training. In another embodiment, approximation engine810 generates an activation level for each sample by causing inferenceengine 800 to perform an inference operation with each sample.Approximation engine 810 transmits training samples 812 to visualizationengine 830 for subsequent incorporation into network analysis GUI 212.

Language engine 820 processes activation data 802 in order to generatedescription 822. Description 822 is a natural language expression thatreflects various high-level characteristics of the operation of neuralnetwork 242 relative to test input 804. For example, description 822could indicate that activation data 802 strongly suggests that testinput 804 should be classified into a particular category. Languageengine 820 can generate natural language descriptions by populating atemplate expression with specific words corresponding to differentactivation levels. For example, a given template could take the form“{adverb} likely to be a {value}.” Language engine 820 could populatethe “adverb” field with different adverbs depending on activation data802. Language engine 820 could also populate the “value” field toindicate a value output by neural network 242 when generating activationdata 802. Language engine 820 transmits description 822 to visualizationengine 830 for subsequent incorporation into network analysis GUI 212.

Visualization engine 830 generates network analysis GUI 212 in order toobtain various information from the user, including test input 804 and aselection of agent 240. For example, network analysis GUI 212 couldreceive user input that should be provided as test input 804 to neuralnetwork 242. Alternatively, network analysis GUI 212 could determine,based on user input, that a particular portion of training data 250should be provided to neural network 242 as test input 804.Visualization engine 830 also updates network analysis GUI 212 toincorporate the various data discussed above, including activation data802, training samples 812, and description 822. Visualization engine 830can also populate network analysis GUI 212 with various other data thatallows the user to inspect the deeper structure of neural network 242,as described in greater detail below in conjunction with FIGS. 9-14 .

FIG. 9 is a screenshot illustrating how the network analysis GUI of FIG.2 facilitates inspection of a neural network, according to variousembodiments. As shown, a GUI panel 900 includes various GUI elementsthat generally relate to the various data discussed above in conjunctionwith FIG. 8 . In particular, input element 902 is a graphical field viawhich inference engine 800 receives test input 804. Selector 904 is aselection field via which inference engine 800 receives a selection ofagent 240. Graph element 906 is a graphical field that displaysactivation data 802. Text element 908 is a text field that displaysdescription 822. Grid element 910 is a graphical field that displaystraining samples 812 within a grid having configurable cells anddimensions.

As also shown, GUI panel 900 includes other GUI elements that depictvarious data associated with neural network 242 and the performance ofneural network 242 relative to test data 804. Specifically, layerelement 920 indicates the different layers of neural network 242 and isconfigured to receive a selection of a particular layer. Metadataelement 930 indicates metadata describing the selected layer. Weightelement 940 includes a grid 932 of weights. Each row in grid 932corresponds to a particular neuron in the selected layer and each columncorresponds to a particular output. A given weight is displayed withvisual attributes reflective of a corresponding weight value. In theexample shown, darker weights have a higher weight value than lighterweights. Weight element 940 is configured to display a natural languagedescription of a selected weight in order to aid the user inunderstanding how the selected weight participates in transforming testinput 804 to produce activation data 802. Activation panel 950 indicatesan activation level 952 associated with the selected layer. In somecases, depending on the selection of layer, activation level 952 may besimilar to activation data 802.

Network analyzer 210 generates the GUI elements described above inconjunction with network analysis GUI 212 in order to expose thefunctionality of neural network 242 to the user and help the user tobuild an intuition regarding how neural network 242 operates undervarious circumstances. This approach differs from conventionaltechniques that do not permit inspection of individual layers, weights,or neurons in the manner described. Accordingly, network analyzer 210provides the user with powerful tools that facilitate rapid developmentof highly accurate neural networks. These techniques can also be appliedin the wider context of agent-based AI models, as described in greaterdetail below in conjunction with FIGS. 10-14 .

FIG. 10 is a screenshot illustrating how the network analysis GUI ofFIG. 2 exposes the underlying functionality of an agent, according tovarious embodiments. As shown, a window 1000 is projected over GUI panel500 of FIG. 5 . Window 1000 exposes the underlying functionality ofagent 240(2) (“read license plates”). Window 1000 is included in networkanalysis GUI 212.

Network analyzer 210 updates network analysis GUI 212 to include window1000 in response to a user selection of agent 240(2). Window 1000includes network architecture 1010, weights 1020, weight metadata 1022,input activation 1030, and output activation 1032. Network architecture1010 is a graphical depiction of the various layers and connectionsbetween layers that define a neural network 242 associated with theselected agent 240(2). Network architecture 1010 is generated similarlyto how network architecture 410 of FIG. 4 is generated.

Network analyzer 210 generates weights 1020 within window 1000 toillustrate the distribution of weight values associated with weightsconnecting adjacent layers in network architecture 1010. Networkanalyzer 210 can display different weights depending on user selectionsof different connections. Network analyzer 210 displays each weight as acell having a particular visual attribute, such as color or shading,that depends on the corresponding weight value. In the example shown,darker weights have greater values than lighter weights. Networkanalyzer 210 also generates weight metadata 1022 to express variousattributes of weights 1020, including the shape of those weights, theminimum weight value, the shape of an associated bias, the minimum valueincluded in that bias, and any other technically feasible attributes ofweights associated with a neural network. Displaying weights in thismanner provides the user with information related to how specific cellsof the neural network favor different outputs, in like fashion asdescribed above in conjunction with weight element 940 of FIG. 9 .

Network analyzer 210 also causes window 1000 to display input activation1030 and output activation 1032 to illustrate how a user-selected layerof the neural network operates in response to a user-selected input. Inparticular, input activation 1030 includes individual cells displayedwith particular visual attributes, such as color or shading, thatindicate the activation level of input connections to the selected layerwhen the neural network processes a selected sample of training data.Additionally, output activation 1032 includes individual cells displayedwith visual attributes that indicate the activation level of outputconnections from the selected layer. Displaying activations in thismanner indicates to the user how the layer transforms an input to anoutput and can help the user understand why the neural network makes (orfails to make) certain decisions.

The techniques described above in conjunction with FIGS. 9-10advantageously can be applied to expose neural network functionality atseveral levels of depth, including network-level functionality,weight-level functionality, and neuron-level functionality, amongothers. Via network analysis GUI 212, network analyzer 210 makesfunctional details of neural networks available to users that are notavailable with conventional approaches. FIGS. 11-14 illustrateadditional situations where the techniques described above can beapplied to inspect and understand neural network operation.

FIG. 11 is a screenshot illustrating how the network analysis GUI ofFIG. 2 exposes a set of agents for processing test inputs, according tovarious embodiments. As shown, a GUI panel 1100 includes a tool panel1110, training data panel 1120, and test input 1122. In the exampleshown, training data panel 1120 includes a set of invoices that need tobe processed to extract various data, including address data, amongothers. Test input 1122 is a sample invoice selected by the user fromtraining data panel 1120. Tool panel 1110 includes a list of differentagents 250 that can be applied to analyze test input 1122. As is shown,tool panel 1110 includes agent 250(5) (“recognize text”), agent 250(6)(“recognize addresses”), agent 250(7) (“select shape”), agent 250(8)(“translate language”), and agent 250(9) (“extract field”). Variousexamples of how network analyzer 210 can apply these agents aredescribed below.

FIG. 12 is a screenshot illustrating how the network analysis GUI ofFIG. 2 applies an agent to process a test input, according to variousembodiments. As shown, based on a user selection of agent 250(5)(“recognize text”), network analyzer 210 updates GUI panel 1100 toemphasize regions of test input 1122 that include text, includingregions 1200, 1210, 1220, 1230, and 1240. Once text is identified inthis manner, additional agents can be applied to perform additionalprocessing tasks, as described in greater detail below.

FIG. 13 is a screenshot illustrating how the network analysis GUI ofFIG. 2 applies another agent to process a test input, according tovarious embodiments. As shown, based on a user selection of agent 250(6)(“recognize addresses”), network analyzer 210 updates GUI panel 1100 toemphasize regions of test input 1122 that include addresses, such asregion 1300. GUI panel 1100 also displays a confidence level with whichthe corresponding region includes an address. In one embodiment, theconfidence level may be derived from a difference in activation levelsassociated with a given layer of the neural network 242 included inagent 250(6). After one or more addresses are identified, another agentcan be applied to extract address data, as described in greater detailbelow.

FIG. 14 is a screenshot illustrating how the network analysis GUI ofFIG. 2 applies a different agent to a test input, according to variousother embodiments. As shown, based on a user selection of agent 250(9)(“extract field”), network analyzer 210 extracts the address from region1300 of test input 1122 and loads that address into an output file 1400.In the example shown, the output file 1400 is a Bill of Lading thatneeds a destination address field to be populated.

Referring generally to FIGS. 11-14 , the example described aboveillustrates how design analysis GUI 212 allows the user to test variousagents 240 on actual input data in order to verify the properfunctionality of those agents. Under circumstances where a given agent240 does not operate as expected, design analysis GUI 212 helps the userto analyze the neural network 242 within the given agent via thetechniques described above in conjunction with FIGS. 9-10 . Variousoperations performed by design analyzer 210 when interacting with theuser via design analysis GUI 212 are described in greater detail belowin conjunction with FIGS. 15A-15B.

FIGS. 15A-15B set forth a flow diagram of method steps for analyzing aneural network via a graphical user interface, according to variousembodiments. Although the method steps are described in conjunction withthe systems of FIGS. 1-2 and 8-14 , persons skilled in the art willunderstand that any system configured to perform the method steps in anyorder falls within the scope of the present embodiments.

As shown in FIG. 15A, a method 1500 begins at step 1502, where networkanalyzer 210 generates network analysis GUI 212 to depict underlyingdata associated with an agent. The agent includes a neural network thatis trained to perform various operations. Network analysis GUI 212depicts various data associated with the neural network, including anetwork architecture, among others.

At step 1504, network analyzer 210 receives a test input to apply to theneural network associated with the agent. For example, network analyzer210 could receive user input describing the test input, such as thehandwritten digit shown in FIG. 9 . Alternatively, network analyzer 210could receive a user selection of a training sample from training data250. Network analyzer 210 generally receives the test input based on oneor more user interactions with network analysis GUI 212.

At step 1506, network analyzer 210 executes an inference operation withneural network based on the test input received at step 1508 to generateactivation data. The activation data could be, for example, activationlevels associated with a specific layer of the neural network. Theactivation data may, in some cases, indicate a probability distributionassociated with a set of classifications the neural network isconfigured to assign to the test input. At step 1508, network analyzer210 updates network analysis GUI 212 to depict the activation data.

At step 1510, network analyzer 210 processes the activation datagenerated at step 1506 to generate a description of the performance ofthe neural network. The description generated by network analyzer 210 isa natural language expression that characterizes at least one functionalor behavioral aspect of the neural network in response to the testinput. For example, the description could indicate that the activationdata indicates a strong likelihood that the neural network can classifythe test input correctly. Network analyzer can generate the descriptionbased on an expression template that is populated with different wordscorresponding to different activation levels and different neuralnetwork outputs. At step 1512, network analyzer 210 updates networkanalysis GUI 212 to depict the description.

At step 1514, based on the activation data generated at step 1508,network analyzer 210 processes training data previously used to trainthe neural network to identify training samples that are similar to thetest input. For example, network analyzer 210 could input each trainingsample to the neural network to generate sample activation data, andthen compare the sample activation data to that generated at step 1508.If the numerical difference between the sample activation data and theactivation data is less than a threshold value, then network analyzer210 would determine that the training sample is similar to the testinput. Persons familiar with neural networks will recognize thatactivation data can include multiple activation levels, and thatcomparing activation data involves comparing corresponding activationlevels. At step 1516, network analyzer 210 updates network analysis GUI212 to depict the training samples. The method 1500 continues in FIG.15B.

At step 1518, network analyzer 210 determines a set of weight valuesassociated with the neural network based on a user interaction networkanalysis GUI 212. For example, network analyzer 210 could receive a userselection of a particular layer of the neural network via networkanalysis GUI 212. Network analyzer 210 could then extract a set ofweight values associated with the layer. The set of weight valuesindicates which neurons contribute, to varying degrees, to whichoutputs. At step 1520, network analyzer 210 updates network analysis GUIto depict the set of weight values. In particular, network analyzer 210generates a grid of cells to represent the set of weight values, whereeach cell is displayed with one or more visual attributes that representthe corresponding weight value.

At step 1522, network analysis GUI 212 determines the output of aselected layer of the neural network in response to an input associatedwith the test input. For example, network analyzer 210 could determineone or more activation levels associated with one or more neurons thatprovide input to the selected layer, and then determine one or moreactivation levels associated with one or more neurons that provideoutput from the selected layer. At step 1524, network analyzer 210updates network analysis GUI to depict the input activation levels andthe output activation levels. In so doing, network analyzer 210 causesnetwork analyzer 212 to display different grids of cells, where eachcell is displayed with a visual attribute that represents acorresponding activation level.

Network analyzer 210 performs the method 1500 in order to provide theuser with detailed information regarding the inner workings of neuralnetworks. This information allows the user to make informed decisionsregarding how to modify the neural network to improve performance. Theneural network can be modified via network generator 200 in the mannerdescribed above in conjunction with FIGS. 3-7 . Network evaluator 220provides additional tools for evaluating the neural network relative tothe training data, as described in greater detail below in conjunctionwith FIGS. 16-27B.

Exploring and Analyzing Data Sets Used to Train Neural Networks

FIGS. 16-27B set forth various techniques implemented by networkevaluator 220 of FIG. 2 when evaluating a neural network relative to thetraining data used to train that neural network. As described in greaterdetail herein, network evaluator 220 generates network evaluation GUI222 in order to facilitate the exploration of the training data based onthe behavior of the neural network.

FIG. 16 is a more detailed illustration of the network evaluator of FIG.2 , according to various embodiments. As shown, network evaluator 220includes an activation engine 1600, a confidence engine 1610, a sortingengine 1620, a saliency engine 1630, and a visualization engine 1640.

In operation, activation engine 1600 receives agent 240 and trainingdata 250 and then executes inference operations with neural network 242across all samples included in training data 250 to generate activationdata 1602. Activation data 1602 includes a set of activation levelsgenerated by neural network 242 for each sample of training data 250. Agiven set of activation levels indicates a probability distributionassociated with a set of categories that neural network can assign tosamples of training data 250. Activation engine 1600 operates similarlyto inference engine 800 of FIG. 8 . Activation engine 1600 transmitstraining data 250 and activation data 1602 to confidence engine 1610,sorting engine 1620, and saliency engine 1630, as well as tovisualization engine 1640 for incorporation into network evaluation GUI222.

Confidence engine 1610 generates confidence data 1612 based on theactivation levels associated with each sample set forth in activationdata 1602. Confidence data 1612 includes a different confidence valuefor each sample that reflects the accuracy with which neural network 242can classify those samples. For a given sample and correspondingactivation levels, confidence engine 1610 determines the differencebetween the greatest activation level (corresponding to a categoryneural network 242 applies to the sample) and one or more otheractivation levels (corresponding to categories neural network 242 doesnot apply to the sample). Accordingly, the confidence value assigned tothe given sample indicates the relative strength with which neuralnetwork 242 assign a category to the sample. In circumstances whereneural network 242 assigns an incorrect category to a sample, the samplecan be labeled “overconfident” indicating that neural network 242strongly indicates an incorrect category for the sample. Confidenceengine 1610 transmits confidence data 1612 to sorting engine 1620 aswell as to visualization engine 1640 for incorporation into networkevaluation GUI 222.

Sorting engine 1620 sorts samples of training data 250 in various waysbased on activation data 1602, confidence data 1612, and user inputreceived via network evaluation GUI 222. In particular, sorting engine1620 groups together samples of training data 250 that are associatedwith similar activation levels included in activation data 1602. Sortingengine 1620 position groups of samples on a two-dimensional map withrelative positions that reflect similarities in activation levels.Sorting engine 1620 also filters samples of training data 250 based oncorresponding confidence values included in confidence data 1612.Sorting engine 1620 generates sorted samples 1622 when performing thesevarious sorting operations and transmits sorted samples 1622 tovisualization engine 1640 for incorporation into network evaluation GUI222.

Saliency engine 1630 processes training data 250 to determine, for anygiven sample of training data 250, the degree to which differentportions of that sample influence the output of neural network 242. Whenprocessing a given sample, saliency engine 1630 applies differentmodifications to one or more portions of the sample to generatedifferent versions of that sample. Saliency engine 1630 then causesneural network 242 to generate separate activation levels based on thedifferent versions of the sample. Saliency engine 1630 compares theactivation levels across the different versions of the sample todetermine whether the modifications to the one or more portions of thesample caused variations in those activation levels. Saliency engine1630 then generates a saliency map that visually indicates the degree towhich various portions of the sample influence the output of neuralnetwork 242. Saliency engine 1630 performs this approach across allsamples of training data 250 to generate saliency data 1632. Saliencyengine 1630 transmits saliency data 1632 to visualization engine 1640for incorporation into network evaluation GUI 222.

Visualization engine 1640 receives training data 250, activation data1602, confidence data 1612, sorted samples 1622, and saliency data 1632and generates and/or updates network evaluation GUI 222 based on thisdata. Network evaluation GUI 222 exposes interactive tools via which theuser can explore training data 250 relative to how neural network 242operates when processing that training data, as described in greaterdetail below in conjunction with FIGS. 17-27B.

FIG. 17 is a screenshot illustrating how network evaluation GUI of FIG.2 facilitates exploration of training data, according to variousembodiments. As shown, a GUI panel 1700 includes a sample map 1710, asample view 1730, activation display 1740, code input 1750, and filterselector 1760. GUI panel 1700 is included in network evaluation GUI 222.The various elements of GUI panel 1700 are described in relation toexemplary training data 250 that includes samples of images that depicthandwritten digits, such as those found in the MNIST digits training setdescribed previously.

Network evaluator 220 generates sample map 1710 via sorting engine 1610described above in conjunction with FIG. 16 . Network evaluator 220generates a different position within sample map 1710 for each sample.The relative positions of any two samples generally reflect thesimilarity of the two samples. Accordingly, samples associated withproximate positions on sample map 1710 are generally similar, andsamples with distant positions on sample 1710 are generally different.Network evaluator 220 can generate sample map 1710 by comparingactivation levels of different samples and then positioning samples withsimilar activation levels within similar regions of sample map 1710 andpositioning samples with different activation levels in differentregions of sample map 1710. Network evaluator 220 can also directlycompare samples of training data 250 to position those samples. In oneembodiment, sample map 1710 may be a t-distributed stochastic neighborembedding (t-SNE) map.

Sample map 1710 includes clusters 1712, 1714, 1716, 1718, and 1720 ofsamples. Each cluster generally corresponds to a particular output ofneural network 242. As such, the activation levels corresponding tosamples associated with a given cluster are generally similar to oneanother. Further, in the example described herein, a given clustergenerally includes samples that depict a specific handwritten digit.Samples are represented in sample map 1710 as either a dot or a cross.Samples represented with a cross are labeled “overconfident” in themanner described previously.

Sample view 1730 displays a graphical depiction of a sample 1732 that isselected via sample map 1710. As is shown, when cursor 1702 ispositioned over a position within cluster 1712, sample view 1730displays a graphical depiction of sample 1732 associated with thatposition. In this instance, a “4” is displayed. Activation display 1740depicts activation levels 1742 associated with sample 1732. Activationlevels 1742 are included in activation data 1602 and generated viaactivation engine 1600 in the manner described above in conjunction withFIG. 16 . Activation levels 1742 indicate that neural network 242provides a strong indication that sample 1732 depicts a “4.” Networkevaluator 220 updates sample view 1730 and activation display 1740 whencursor 1702 is moved within sample map 1710, as is shown in FIG. 18 .

FIG. 18 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 receives input via a sample map, according to variousembodiments. As shown, when cursor 1702 is positioned over a positionwithin cluster 1714, sample view 1730 displays a graphical depiction ofsample 1832 associated with that position. In this instance, a “3” isdisplayed. Activation display 1740 depicts activation levels 1842associated with sample 1832, which indicate that neural network 242provides a moderate indication that sample 1832 depicts a “3.”

Referring generally to both FIGS. 17 and 18 , code input 1750 is a textfield via which the user can write program code for processing andfiltering sample map 1710. The example code shown causes networkanalyzer 220 to assign a different color to each cluster of samples whengenerating sample map 1710. Code input 1750 can be pre-populated withprogram code generated by network evaluator 220. Filter selector 1760 isan input element that receives user input indicating a particular filterto apply to sample map 1710. Each filter generally corresponds to aportion of program code that, when executed, modifies sample map 1710.Upon selection of a given filter via filter selector 1760, networkevaluator 220 populates code input 1750 with the portion of program codecorresponding to that filter, thereby allowing the user to customize andexecute that program code. Various examples of how network evaluator 220can modify sample map 1710 are described below in conjunction with FIGS.19-21 .

FIG. 19 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data assigned a high confidencevalue, according to various embodiments. As shown, in response to a userselection of a “high confidence” filter, network evaluator 220 updatessample map 1710 to only display positions corresponding to samplesassigned a high confidence value. Network evaluator 220 assignsconfidence values to samples via confidence engine 1610 described abovein conjunction with FIG. 16 . As previously discussed, the confidencevalue assigned to a given sample represents the difference between thehighest activation level associated with the sample and one or moreother activation levels.

In the example shown, cursor 1702 resides at a position within cluster1716 associated with sample 1932, which depicts a “2.” Activation levels1942 indicate that neural network 242 provides a very strong indicationthat sample 1932 depicts a “2.” Since neural network 242 does notprovide any other significant indications, sample 1932 is assigned ahigh confidence value and is therefore shown when sample map 1710 isfiltered in the manner discussed.

Code input 1750 includes program code that is executed via networkanalyzer 220 to identify samples with high confidence values and to thenupdate sample map 1710 to only display those samples. Network analyzer220 can receive modifications to the code shown in code input 1750 andthen execute the modified code to update sample map 1710. For example,network analyzer 220 could receive a modification to a thresholdconfidence value and then cause sample map 1710 to display samples withconfidence values that exceed the modified threshold confidence level.Network analyzer 220 can also filter samples with other filters, asdescribed below in conjunction with FIGS. 20-21 .

FIG. 20 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data assigned a low confidencevalue, according to various embodiments. As shown, in response to a userselection of a “low confidence” filter, network evaluator 220 updatessample map 1710 to only display positions corresponding to samplesassigned a low confidence value. As also shown, cursor 1702 resides at aposition within cluster 1718 associated with sample 2032, which depictsa “5.” Activation levels 2042 indicate that neural network 242 providesa weak indication that sample 2032 depicts a “5” and a weak indicationthat sample 2032 depicts a “2.” Since neither indication greatly exceedsthe other, sample 2032 is assigned a low confidence value and istherefore shown when sample map 1710 is filtered in the mannerdiscussed.

FIG. 21 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data labeled overconfident,according to various embodiments. As shown, in response to a userselection of an “overconfident” filter, network evaluator 220 updatessample map 1710 to only display positions corresponding to sampleslabeled “overconfident.” These samples may have a negative confidencevalue. As also shown, cursor 1702 resides at a position within cluster1714 associated with sample 2132, which depicts a “3.” Activation levels2142 indicate that neural network 242 provides a strong indication thatsample 2132 depicts a “2” and a weak indication that sample 2132 depictsa “3.” Since neural network 242 provides an incorrect output relative tosample 2132, sample 2132 is labeled “overconfident” and is thereforeshown when sample map 1710 is filtered in the manner discussed.

As a general matter, network evaluator 220 can perform the evaluationtechniques described above based on any technically feasible set oftraining data 250 beyond the exemplary training data discussed inconjunction with FIGS. 17-21 . FIGS. 22-27 depict how network evaluator220 performs other evaluation techniques relative to another exemplaryset of training data.

FIG. 22 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 indicates samples of training data that promote a selected neuralnetwork output, according to various embodiments. As shown, an updatedversion of window 1000 of FIG. 10 includes input activation 1030 andoutput activation 1032, with other elements of window 1000 omitted forclarity. As previously discussed, output activation 1032 includes a gridof cells that correspond to an output of a selected layer of a neuralnetwork when processing a sample of training data 250 included intraining data panel 530.

Upon selection of a cell 2200 within output activation 1032, networkevaluator 220 emphasizes specific samples within training data 250 thatcause cell 2200 to provide an elevated output. As is shown, networkevaluator 220 emphasizes samples 2202 and 2204, indicating that cell2200 provides an elevated output when neural network 224 processessamples 2202 and 2204. An advantage of this technique is that the usercan gain insight into how the neurons within specific layers of neuralnetwork 224 respond to different types of samples included in trainingdata 250. Network evaluator 220 can also sort training data 250 based ona selected cell, described in greater detail below in conjunction withFIG. 23 .

FIG. 23 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data sorted based on a neuralnetwork output, according to various embodiments. As shown, in responseto the user selection of cell 2200, network evaluator 220 sorts trainingdata 250 to place samples that promote activation of the neuronassociated with cell 2200 towards the left side of training data panel530 and to place samples that do not promote activation of the neuronassociated with cell 2200 on the right side of training data panel 530.

In addition, network evaluator 220 generates activation panel 2300 thatincludes a graph 2303. Graph 2302 indicates how strongly differentportions of the sorted training data promote activation of the neuronassociated with cell 2200. For example, graph 2302 has an elevated levelabove samples 2202 and 2204, but tapers down from left to right inconjunction with samples that promote the activation of the neuron to alesser degree. Network evaluator 220 can perform the techniquesdescribed above in conjunction with FIGS. 22-23 relative to anexpression that relates the outputs of multiple neurons, as described ingreater detail below in conjunction with FIGS. 24-25 .

FIG. 24 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 indicates samples of training data that meet specific activationcriteria, according to various embodiments. As shown, expression input2400 includes a conditional expression against which samples included intraining data 250 are tested. In particular, for a given trainingsample, network evaluator 220 determines the activation level of eachneuron included in the expression when neural network 224 processes thegiven training sample. Network evaluator 220 then evaluates theexpression based on the determined activation levels to output atrue/false value. Network evaluator 220 emphasizes the specific samplesfor which the conditional expression evaluates to logical true. In theexample shown, the expression evaluates affirmatively for samples 2402and 2404, and so network evaluator 220 emphasizes those samples. Networkevaluator 220 can also sort samples of training data 250 based onuser-generated expressions, as described below in conjunction with FIG.25 .

FIG. 25 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays samples of training data sorted based on an expression,according to various embodiments. As shown, expression input 2400includes an arithmetic expression based on which samples included intraining data 250 are sorted. For a given training sample, networkevaluator 220 determines the activation level of each neuron included inthe expression when neural network 224 processes the given trainingsample. Network evaluator 220 evaluates the expression based on thedetermined activation levels to generate an output value. Networkevaluator 220 then sorts training data 250 based on the output valuesassociated with each sample. In the example shown, samples 2402 and 2404are associated with elevated output values, and so network evaluator 220sorts those samples to the left side of training data panel 530. Networkevaluator 220 also generates graph 2502 within activation panel 2300 toindicate the output levels associated with corresponding samples oftraining data 250.

Referring generally to FIGS. 22-25 , network evaluator 220 performs thedisclosed sorting techniques via sorting engine 1620 describedpreviously in conjunction with FIG. 16 . Saliency engine 1630 of FIG. 16performs an additional technique for determining specific portions oftraining data samples influence the output of neural network 224, asdescribed in greater detail below in conjunction with FIG. 26 .

FIG. 26 is a screenshot illustrating how the network evaluation GUI ofFIG. 2 displays relevant portions of a training sample, according tovarious embodiments. As shown, saliency display 2600 includes a saliencymap 2602 of a selected sample 2604. Saliency map 2602 indicates specificportions of sample 2604 that influence changes in the output of neuralnetwork 224 in response to sample 2604. Network evaluator 220 generatessaliency map 2602 by performing a sensitivity analysis with sample 2604.In doing so, network evaluator 220 generates slightly modified versionsof sample 2604 and then determines how the output of neural network 224changes relative to those slightly modified versions. Network evaluator220 then assigns a sensitivity value to each portion of sample 2604indicating the degree to which that portion affects the output of neuralnetwork 224. In the example shown, the front portion of the automobiledepicted in sample 2604 is shaded to indicate that changes to the frontportion of the automobile lead to changes in the output of neuralnetwork 224.

Referring generally to FIGS. 16-26 , the disclosed techniques providethe user with a range of tools for evaluating a neural network relativeto training data based on which the neural network is trained. Personsskilled in the art will understand that the disclosed techniques can beapplied evaluate neural networks based on any set of data, beyond thetraining data used to train the neural network. The techniques performedby network evaluator 220 described thus far are described in greaterdetail below in conjunction with FIGS. 27A-27B.

FIGS. 27A-27B set forth a flow diagram of method steps for evaluating aneural network relative to a set of training data via a graphical userinterface, according to various embodiments. Although the method stepsare described in conjunction with the systems of FIGS. 1-2 and 16-26 ,persons skilled in the art will understand that any system configured toperform the method steps in any order falls within the scope of thepresent embodiments.

As shown in FIG. 27A, a method 2700 begins at step 2702, where networkevaluator 220 obtains samples of training data used to train a neuralnetwork. In various embodiments, network evaluator 220 may also obtainsamples of training data not used to train the neural network, such assamples included within a validation set. Network evaluator 220 performsvarious techniques for evaluating the neural network relative to theobtained training data.

At step 2704, network evaluator 220 generates activation data forsamples that includes activation levels for each sample. For example,network evaluator 220 could input each sample to the neural network andthen record the output of a particular layer of the neural network, suchas the second-to-last layer. The set of activation levels for a giventraining sample specifies how strongly the neural network indicates eachpossible output for the associated sample.

At step 2706, network evaluator 220 generates a confidence value foreach sample based on the corresponding set of activation levels. For agiven sample and corresponding activation levels, network evaluator 220determines the difference between the greatest activation level and oneor more other activation levels. Conceptually, the confidence valueassigned to a given sample indicates the relative strength with whichthe neural network classifies the sample.

At step 2708, network evaluator 220 groups samples based on theactivation levels generated at step 2704. For example, network evaluator220 could compare the activation levels associated with two samples andassign a difference value to that pair of samples. Network evaluator 220could then collect samples with low mutual difference values into aparticular group. When comparing two sets of activation levels, networkevaluator 220 generally compares activation levels associated with thesame classification.

At step 2710, network evaluator 220 generates network evaluation GUI 222to display groups of samples, activation levels, and confidence values.In so doing, network evaluator 220 causes network evaluation GUI 222 todisplay a sample map indicating the groups of samples generated at step2708. An exemplary sample map is depicted in FIG. 17 . Network evaluator220 also causes network evaluation GUI 222 to display the activationlevels generated at step 2704 and, in some embodiments, the confidencevalues generated at step 2706.

At step 2712, network evaluator 220 receives a selection of filtrationcriteria that should be used to filter the display of data associatedwith samples of training data. A given filtration criteria couldindicate, for example, that only samples assigned elevated confidencevalues should be displayed. In response to the selected filtrationcriteria, at step 2714, network evaluator 220 updates network evaluationGUI 222 to modify one or more groups of samples based on the assignedconfidence values and the filtration criteria received at step 2712. Inparticular, network evaluator 220 causes network evaluation GUI 222 toonly display data associated with samples that meet the filtrationcriteria. The method 2700 continues in FIG. 27B.

At step 2716, network evaluator 220 receives a selection of an outputneuron associated with the neural network. The output neuron can residein any of the layers of the neural network. In practice, networkevaluator 220 receives a selection of a given layer from the user, andthen network evaluator 220 receives a selection of a particular outputassociated with that layer.

At step 2718, network evaluator 220 sorts samples of the training databased on the activation levels generated at step 2704 and based on theactivation level of selected neuron. In particular, network evaluator220 ranks the samples relative to how closely the activation levelsassociated with the samples match the activation level associated withthe selected neuron, thereby indicating the specific samples thatstrongly promote activation of the selected neuron. At step 2720,network evaluator 220 updates network evaluation GUI 222 to display thesorted samples. In so doing, network evaluator 220 can generate a graphindicating the degree to which each sample promotes the activation ofthe selected neuron.

At step 2722, network evaluator 220 receives an expression that relatesactivation levels of a set of neurons. The expression could be aconditional expression that evaluates to true or false, or an arithmeticexpression that evaluates to a numerical value. Network evaluator 220evaluates the expression based on the activation level produced by theneural network in response to each sample. Network evaluator 220 assignsthe result of that evaluation to the corresponding sample.

At step 2724, network evaluator 220 sorts the samples of training databased on evaluation of the expression. For example, network evaluator220 could identify the specific samples for which the expressionevaluates to true. At step 2726, network evaluator 220 updates networkevaluation GUI 222 to display the sorted samples. In so doing, networkevaluator 220 can generate a graph indicating the result of evaluatingthe expression for each sample.

At step 2728, network evaluator 220 generates a saliency map thatindicates regions of a selected sample that influence the output of theneural network. Network evaluator 220 generates the saliency map byperforming a sensitivity analysis with sample. Specifically, networkevaluator 220 generates slightly modified versions of each sample andthen determines how the output of the neural network changes relative tothose slightly modified versions. At step 2730, network evaluator 220updates network evaluation GUI 222 to display the saliency map.

Referring generally to FIGS. 16-27B, network evaluator 220advantageously provides techniques for analyzing and evaluating how aneural network operates relative to training data, thereby allowing theuser to gain insight and intuition into how to improve the operation ofthe neural network. Additionally, network evaluation GUI 222 facilitatesthe user in analyzing and exploring training data based on how theneural network responds to the training data, thereby assisting the userin furthering that intuition. Network descriptor 230 described above inconjunction with FIG. 2 performs additional that can be applied todescribe and constrain the performance neural networks, as described ingreater detail below in conjunction with FIGS. 28-38B.

Articulating and Constraining the Behavior of Neural Networks

FIGS. 28-38B set forth various techniques implemented by networkdescriptor 230 of FIG. 2 when analyzing the behavior of a neuralnetwork. As described in greater detail herein, network descriptor 230generates network description GUI 232 to express various data thatdescribed the behavior of the neural network and to constrain thatbehavior based on user input.

FIG. 28 is a more detailed illustration of the network descriptor ofFIG. 2 , according to various embodiments. As shown, network descriptor230 includes a rules engine 2800, an articulation engine 2810, aperformance engine 2820, and a visualization engine 2830.

In operation, rules engine 2802 analyzes the behavior of a set ofneurons within neural network 242 when processing training data 250 andgenerates rules 2802 for modifying the output of neural network 242. Forexample, a given rule included in rules 2802 could indicate that when agiven neuron included in a given layer of neural network 242 outputs acertain value, that the output of neural network 242 is inaccurate andshould be replaced with an alternate output. Rules engine 2802 cangenerate rules automatically based on the performance of neural network242 when processing training data 250 by identifying specific patternsof neuron activity that occur when neural network 242 produces incorrectoutputs. Rules engine 2802 labels these specific patterns as “specialcases” and generates alternative outputs for these special cases. Rulesengine 2802 can also receive user input via network description GUI 232indicating specific rules that should be applied to, or integrated into,neural network 242. Rules engine 2800 can also expose rules 2802 to theuser via network description GUI 232 for modification. Rules engine 2800transmits rules 2802 to visualization engine 2830 for incorporation intonetwork description GUI 232. The operation of rules engine 2802 isdescribed in greater detail below in conjunction with FIG. 29 .

Articulation engine 2810 analyzes the behavior neural network 242 whenprocessing training data 250 and generates articulated knowledge 2812that describes various characteristics of neural network 242 via naturallanguage expressions. For example, articulation engine 2810 can analyzethe accuracy of neural network 242 across a range of samples of trainingdata 250 and then generate a natural language expression indicating theparticular types of samples that neural network can classify mostaccurately. Articulation engine 2810 can also generate articulatedknowledge 2812 based on data stored in knowledge base 2850. Knowledgebase 2850 includes logical facts that articulation engine 2810 maps tovarious behaviors of neural network 242 when processing specific samplesof training data 250. For example, suppose neural network 242 classifiesa sample of training data 250 as depicting a car that includes a door.Articulation engine 2810 could extract a logical fact from knowledgebase 2850 indicating that the side of a car has a door. Based on thislogical fact, articulation engine 2810 could generate articulatedknowledge 2812 indicating that the sample of training data 250 depictsthe side of the car. Articulation engine 2810 transmits articulatedknowledge 2812 to visualization engine 2830 for incorporation intonetwork description GUI 232. The operation of articulation engine 2810is described in greater detail below in conjunction with FIGS. 30-31 .

Performance engine 2820 analyzes the performance of neural network 242during training and when subsequently performing inference operationsand generates performance data 2822 that quantifies the performance ofneural network 242. In particular, performance data 2822 indicates howquickly neural network 242 converges to various levels of accuracy, howquickly neural network 242 can classify different inputs, and how muchmemory each layer of neural network 242 consumes during execution.Performance engine 2820 can also generate alternate versions of neuralnetwork 242 and perform a comparative analysis of these alternateversions. Performance engine 2820 transmits performance data 2822 tovisualization engine 2830 for incorporation into network description GUI232. The operation of performance engine 2822 is described in greaterdetail below in conjunction with FIGS. 32-37 .

Visualization engine 2830 receives rules 2802, articulated knowledge2812, and performance data 2822 and generates and/or updates networkdescription GUI 232 based on this data. Network description GUI 232exposes interactive tools via which the user can generate and/or modifyrules 2802, view articulated knowledge 2812, generate performance 2822,and analyze alternative versions of neural network 242, as described ingreater detail below in conjunction with FIGS. 29-37 .

FIG. 29 is a screenshot illustrating how the network description GUI ofFIG. 2 facilitates the constraining of neural network behavior undervarious circumstances, according to various embodiments. As shown, rulesinput 2900 includes a rule 2902 that specifies circumstances under whichneural network 242 should generate a modified output data. Inparticular, rule 2902 includes program code indicating that if theactivation data is considered a special case, then special case outputdata 2912 should be output instead of output data 2910. The activationdata could include, for example, the outputs of one or more neuronswithin one or more layers of neural network 242 or an expression that isbased on those outputs and evaluates to a given value. When neuralnetwork 242 performs inference operations, the program code associatedwith rule 2902 is executed in order to identify special case situationsand to modify the output of neural network 242 in response.

Network descriptor 230 can generate program code for rule 2902automatically by analyzing activation patterns of neural network 242when generating incorrect outputs and then mapping those activationpatterns to correct outputs. Network descriptor 230 can also receiveprogram code defining a rule 2902 from the user via rule input 2900. Inaddition to generating rules that constrain network behavior, networkdescriptor can also generate expressions that describe network behavior,as described in greater detail below in conjunction with FIGS. 30-31 .

FIG. 30 is a screenshot illustrating how the network description GUI ofFIG. 2 articulates neural network behavior, according to variousembodiments. As shown, articulation panel 3000 includes vocabulary 3002,definition 3004, common sense facts 3006, and derived facts 3008.Articulation panel 3000 is included in network description GUI 232.

Network descriptor 230 obtains vocabulary 3002, definitions, 3004, andcommon sense facts 3006 from knowledge base 2850. Vocabulary 3002includes various terms that are associated with cars. Definitions 3004include definitions of terms that are associated with cars. Common sensefacts 3006 include logical facts that are generally applicable, andother logical facts that are specifically applicable to automobiles.Network descriptor 230 generates derived facts 3008 based on thebehavior of neural network 242 when analyzing a sample of training data250. In the example described herein, the sample of training data 250 isan image of a car, as shown in segmentation panels 3010, 3012, 3014, and3016.

Segmentation panels 3010, 3012, 3014, and 3016 depict varioussegmentation maps that neural network 242 generates based on the sampleof training data 250. Segmentation panel 3010 indicates regions of thesample that are associated with a car. Segmentation panel 3012 indicatesregions of the sample that are associated with the wheels of the car.Segmentation panel 3014 indicates regions of the sample that areassociated with the back of the car. Segmentation panel 3016 indicatesregions of the sample that are associated with the rear license plate ofthe car.

Network descriptor 230 generates derived facts 3008 by logicallycombining common sense facts 3006 based on the segmentation mapsgenerated for the sample of training data 250. Network descriptor 230can reveal the logical process used to generate each derived fact 3008in response to user input, as described below in conjunction with FIG.31 .

FIG. 31 is a screenshot illustrating how the network description GUI ofFIG. 2 represents a derived fact, according to various embodiments. Asshown, articulation panel 3000 includes explanation 3100 the outlinesthe logical steps network descriptor 230 implements to determine thatthe car in the sample of training data 250 is facing away. Inparticular, network descriptor 230 determines that neural network 242identified a trunk in the sample of training data 250, as shown insegmentation panel 3014. Network descriptor 230 also determines thatbecause most cars have a trunk on the back, that the back of the car isvisible. Network descriptor 230 also determines that when the back ofsomething is visible, that thing is facing away, as set forth in commonsense facts 3006. Based on these various facts, network descriptor 230concludes that the car shown in the sample is facing away.

Referring generally to FIGS. 30-31 , network descriptor 230advantageously provides natural language descriptions and explanationsthat characterize how neural network 242 performs when processingdifferent inputs. Based on these explanations, the user can develop agreater understanding of how neural network 242 performs and whetherneural network 242 operates suitably for various tasks. Networkdescriptor 230 also generates performance data that quantifies howneural network 242 performs during training and inference, as describedin greater detail below in conjunction with FIGS. 32-37 .

FIG. 32 is a screenshot illustrating how the network description GUI ofFIG. 2 depicts performance data associated with the training of a neuralnetwork, according to various embodiments. As shown, a performance panel3200 includes network architecture 3202 that is associated with neuralnetwork 242 of FIG. 28 and an accuracy graph 3210. Network architecture3202 is an interactive GUI element that is configured to modify theunderlying definition of neural network 242 in response to user input,as previously described. Accuracy graph 3210 includes plot 3212 thatrepresents how the accuracy of neural network 242 changes over timeduring training. As is shown, the accuracy with which neural network 242performs operates improves over time during the training procedure.Network descriptor 230 generates performance panel 330 to assist theuser with evaluating neural network 242 and also generates other typesof performance panels that are described in greater detail below.

FIG. 33 is a screenshot illustrating how the network description GUI ofFIG. 2 depicts other performance data associated with the training of aneural network, according to various other embodiments. As shown, aperformance panel 3300 includes network architecture 3302 associatedwith neural network 242 of FIG. 28 and an inference graph 3310.Inference graph 3310 includes plot 3312 that indicates the inferencetime needed to classify different samples of training data. As is shown,neural network 242 needs different amounts of time to process differentsamples 3320.

Referring generally to FIGS. 32-33 , network descriptor 230 generatesthe performance data described in conjunction with these figures todescribe the performance of neural network 242 during operation. Networkdescriptor 230 also captures data indicating the amount of computationalresources consumed when neural network 242 executes, as described ingreater detail below.

FIG. 34 is a screenshot illustrating how the network description GUI ofFIG. 2 displays the amount of memory consumed when executing a neuralnetwork, according to various embodiments. As shown, resources panel3400 includes network architecture 3402 and memory chart 3410. Memorychart 3410 is a bar graph indicating the amount of memory that isconsumed during execution of each layer set forth in networkarchitecture 3402. The second convolution layer consumes the most memoryat 144 kilobytes. Memory chart 3410 can also indicate the total amountof memory consumed when neural network 242 executes.

Network descriptor 230 generates the various panels described above inconjunction with FIGS. 32-34 to provide the user with valuable insightinto how neural network 242 operates. Based on this information, theuser can decide whether neural network 242 needs to be modified. Networkdescriptor 230 generates additional panels that allow the user togenerate and test alternate versions of neural network 242, as describedbelow in conjunction with FIGS. 35-37 .

FIG. 35 is a screenshot illustrating how the network description GUI ofFIG. 2 represents different versions of a given neural network,according to various embodiments. As shown, modification panel 3500includes network architecture 3502 with which the user can interact togenerate alternate network architectures. For example, the user couldinteract with modification element 3504 to increase or decrease the sizeof a given layer included in network architecture 3502. Alternateversion panels 3510 and 3520 depict alternate network architectures 3512and 3522, respectively, that are generated based on user modificationsto network architecture 3502. Network descriptor 230 can perform acomparative analysis with these different versions of neural network 242to generate additional performance data, as described in greater detailbelow.

FIG. 36 is a screenshot illustrating how the network description GUI ofFIG. 2 displays comparative performance data associated with differentversions of a given neural network, according to various embodiments. Asshown, comparative performance panel 3600 includes alternate networkarchitectures 3512 and 3522 as well as accuracy graph 3610. Accuracygraph 3610 includes plots 3612 and 3622 that represent the accuracy ofthe different versions of neural network 242 during training. Plot 3612corresponds to network architecture 3512 and plot 3622 corresponds tonetwork architecture 3522. As is shown, network architecture 3512achieves a high degree of accuracy faster than network architecture3522. Network descriptor 230 provides the user with additional datacharacterizing alternate versions of neural network 242, as described ingreater detail below.

FIG. 37 is a screenshot illustrating how the network description GUI ofFIG. 2 displays other comparative performance data associated withdifferent versions of a given neural network, according to various otherembodiments. As shown, comparison panel 3700 includes alternate networkarchitectures 3512 and 3522 as well as comparison panels 3712 and 3722corresponding to those network architectures. Comparison panels 3712 and3722 convey various performance data associated with the respectivenetwork architectures, thereby allowing the user to evaluate whether themodifications made to neural network 242 increase or decreaseperformance.

Referring generally to FIGS. 32-37 , network descriptor 230 generatesand/or updates network description GUI 232 with the various panelsdescribed in conjunction with these figures to provide the user withinformative data that can assist the user with improving neural network242. Advantageously, the various tools exposed via network descriptionGUI 232 provide convenient mechanisms via which the user can generateand modify neural networks.

Network descriptor 230 in general provides a broad range of operationsfor describing various aspects of neural network behavior,characterizing and quantifying neural network behavior, and constrainingneural network behavior under specific circumstances. The operation ofnetwork descriptor 230 is described in greater detail below inconjunction with FIGS. 38A-38B.

FIGS. 38A-38B set forth a flow diagram of method steps for articulatingand constraining the behavior of a neural network via a graphical userinterface, according to various embodiments. Although the method stepsare described in conjunction with the systems of FIGS. 1-2 and 28-37 ,persons skilled in the art will understand that any system configured toperform the method steps in any order falls within the scope of thepresent embodiments.

As shown in FIG. 38A, a method 3800 begins at step 3802 where networkdescriptor 230 of FIG. 2 obtain samples of training data used to train aneural network. The samples of training data can include any technicallyfeasible dataset, including, for example, a set of images of handwrittendigits, a set of images of automobiles, a set of audio files, and soforth.

At step 3804, network descriptor 230 generates activation data for asample in the training data. For example, network descriptor 230 couldcause the neural network to perform an inference operation with thesample of training data to generate a classification for that sample.Network descriptor 230 could then analyze the output of a set of neuronsassociated with a given layer of the neural network to generate theactivation data.

At step 3806, network descriptor 230 determines an output of the neuralnetwork in response to the sample of training data. For example, networkdescriptor 230 could determine a classification that the neural networkassigns to the sample of training data. The output may not necessarilybe correct. However, network descriptor 230 can modify the output of theneural network to correct incorrect outputs based on the activation datagenerated at step 3804.

At step 3808, network descriptor 230 generates a rule that modifies theoutput of the neural network based on the activation data. Undercircumstances where the neural network exhibits activation patters thatare consistent with the activation data, the rule is applied to causethe neural network to generate a modified output. FIG. 29 includes anexample of a rule that can be applied to modify the output of the neuralnetwork. Network descriptor 230 implements the above steps in order toconstrain the behavior of the neural network. Network descriptor 230also implements the following steps to articulate the behavior of theneural network.

At step 3810, network descriptor 230 determines a set of domain factsthat are relevant to the training data used to train the neural network.The set of domain facts can be derived from a knowledge base thatincludes logical facts that are specifically applicable to the trainingdata. For example, a set of domain facts associated with automobilescould indicate that most cars have four wheels or that the back of a cartypically has a trunk.

At step 3812, network descriptor 230 determines a set of generalknowledge facts. The set of general knowledge facts can be derived froma knowledge base that includes generally applicable facts that may berelevant in a wide variety of contexts. For example, network descriptor230 could determine a general knowledge fact indicating that if the backof something is visible then the thing is facing away from the viewer.

At step 3814, network descriptor 230 compares the set of domain facts tothe set of general knowledge facts to generate one or more derivedfacts. For example, network descriptor 230 could generate a derived factindicating that a particular sample includes an automobile that isfacing away because the trunk of the car is visible, and the generalknowledge fact indicates that when the back of something is visible thenthat thing is facing away. Network descriptor 230 can apply thisapproach to any technically feasible type of training data beyond thatassociated with automobiles. At step 3816, network descriptor 230updates network description GUI 232 to display the set of domain facts,the set of general knowledge facts, and the one or more derived facts.The method 3800 continues in FIG. 38B.

At step 3818, network descriptor 230 generates one or more differentversions of the neural network. For example, network descriptor 230could receive a user modification to a given layer of the neural networkvia a graphical depiction of the network architecture associated withthe neural network. In this manner, network descriptor 230 allows theuser to generate and test variations of the neural network in order toidentify changes that improve the performance of the neural network.

At step 3820, network descriptor 230 generates performance data for eachversion of the neural network. For a given version of the neuralnetwork, the performance data can indicate how the accuracy of theneural network changes during training, how much time the neural networkneeds to perform inference operations with different samples of trainingdata, how much memory each layer of the neural network consumes, andother data that characterizes the performance of the neural network. Atstep 3822, network descriptor 230 updates network description GUI 232 todisplay the performance data, as also described by way of example abovein conjunction with FIGS. 32-37 .

Via the above techniques, network descriptor 330 can both articulatenatural language descriptions that characterize the behavior of a neuralnetwork and constrain that behavior to increase neural network accuracy.Accordingly, these techniques empower the user to develop a greaterunderstanding of how the neural network operates, to communicate thatunderstanding to others, and to modify the output of the neural networkas needed.

In sum, an artificial intelligence (AI) design application that exposesvarious tools to a user for generating, analyzing, evaluating, anddescribing neural networks. The AI design application includes a networkgenerator that generates and/or updates program code that defines aneural network based on user interactions with a graphical depiction ofthe network architecture. The AI design application also includes anetwork analyzer that analyzes the behavior of the neural network at thelayer level, neuron level, and weight level in response to test inputs.The AI design application further includes a network evaluator thatperforms a comprehensive evaluation of the neural network across a rangeof sample of training data. Finally, the AI design application includesa network descriptor that articulates the behavior of the neural networkin natural language and constrains that behavior according to a set ofrules.

At least one technological advantage of the disclosed techniquesrelative to the prior art is that the disclosed AI design applicationcan generate complex neural network architectures without requiring adesigner user to write or interact with large amounts of program code.Another technological advantage of the disclosed techniques relative tothe prior art is that the disclosed AI design application provides adesigner with detailed information about the underlying operations andfunctions of the individual components of a given neural networkarchitecture. Accordingly, the AI design application enables a designerto develop and better understanding of how the neural network operates.Another technological advantage of the disclosed techniques relative tothe prior art is that the disclosed AI design application performsdetailed analyses of how a given neural network operates during thetraining phase, thereby enabling a designer to better understand why theneural network generates specific outputs based on particular inputs.Yet another technological advantage of the disclosed techniques relativeto the prior art is that the disclosed AI design applicationautomatically generates natural language descriptions characterizing howa given neural network operates and functions. Among other things, thesedescriptions help explain the operations of the neural network to adesigner and enable the designer to articulate and explain thefunctional characteristics of the neural network to others. Thesetechnological advantages represent one or more technologicaladvancements over prior art approaches.

1. Some embodiments include a computer-implemented method for analyzinghow a neural network has been trained, the method comprising causing theneural network to execute an inference operation based on a plurality ofsamples of training data to generate a plurality of portions ofactivation data, wherein each sample of training data corresponds to adifferent portion of activation data, generating a position value foreach sample of training data based on the portion of activation datacorresponding to the sample of training data, and generating a graphicaluser interface that displays each sample of training data positionedwithin the graphical user interface based on the position valuegenerated for the sample of training data.

2. The computer-implemented method of clause 1, wherein generating theposition value for each sample of training data comprises performing adimensionality reduction operation based on the plurality of portions ofactivation data.

3. The computer-implemented method of any of clauses 1-2, wherein agiven portion of activation data comprises an N-dimensional vector,wherein N is a positive integer greater than two, and wherein thegraphical user interface includes a two-dimensional projection of theplurality of portions of activation data.

4. The computer-implemented method of any of clauses 1-3, whereingenerating the graphical user interface comprises generating at-distributed stochastic neighbor embedding (t-SNE) map based on theplurality of portions of activation data.

5. The computer-implemented method of any of clauses 1-4, whereingenerating the graphical user interface comprises ranking each sample oftraining data based on the position value generated for the sample oftraining data to produce a plurality of ranked samples of training data,and positioning each ranked sample of training data within a grid.

6. The computer-implemented method of any of clauses 1-5, furthercomprising generating a confidence value for each sample of trainingdata based on at least one activation level included in the portion ofactivation data corresponding to the sample of training data, receivinga selection of a first confidence criterion via the graphical userinterface, determining a subset of samples of training data havingconfidence values that meet the first confidence criterion, and updatingthe graphical user interface to remove all samples of training data thatare not included in the subset of samples of training data.

7. The computer-implemented method of any of clauses 1-6, whereindetermining the subset of samples of training data comprises identifyingat least one sample of training data having a confidence value that isgreater than a confidence threshold corresponding to the confidencecriterion.

8. The computer-implemented method of any of clauses 1-7, whereindetermining the subset of samples of training data comprises identifyingat least one sample of training data having a confidence value that isless than a confidence threshold corresponding to the confidencecriterion.

9. The computer-implemented method of any of clauses 1-8, whereindetermining the subset of samples of training data comprises identifyingat least one sample of training data corresponding to a portion ofactivation data that indicates an incorrect output of the neural networkwhen the inference operation is executed.

10. The computer-implemented method of any of clauses 1-9, furthercomprising generating a confidence value for each sample of trainingdata by determining a difference value between a first activation levelincluded in the portion of activation data corresponding to the sampleof training data and a second activation level included in the portionof activation data corresponding to the sample of training data, andupdating the graphical user interface to display each sample of trainingdata based on the confidence value generated for the sample of trainingdata.

11. Some embodiments include a non-transitory computer-readable mediumstoring program instructions that, when executed by a processor, causethe processor to analyze how a neural network has been trained byperforming the steps of causing the neural network to execute aninference operation based on a plurality of samples of training data togenerate a plurality of portions of activation data, wherein each sampleof training data corresponds to a different portion of activation data,and generating a graphical user interface that displays each sample oftraining data positioned within the graphical user interface based on aposition value generated for each sample of training data based on theportion of activation data corresponding to the sample of training data.

12. The non-transitory computer-readable medium of clause 11, whereingenerating the position value for each sample of training data comprisesperforming a dimensionality reduction operation based on the pluralityof portions of activation data.

13. The non-transitory computer-readable medium of any of clauses 11-12,wherein a given portion of activation data comprises an N-dimensionalvector, wherein N is a positive integer greater than two, and whereinthe graphical user interface includes a two-dimensional projection ofthe plurality of portions of activation data.

14. The non-transitory computer-readable medium of any of clauses 11-13,wherein the step of generating the graphical user interface comprisesgenerating a t-distributed stochastic neighbor embedding (t-SNE) mapbased on the plurality of portions of activation data.

15. The non-transitory computer-readable medium of any of clauses 11-14,wherein the step of generating the graphical user interface comprisesranking each sample of training data based on the position valuegenerated for the sample of training data to produce a plurality ofranked samples of training data, and positioning each ranked sample oftraining data within a grid.

16. The non-transitory computer-readable medium of any of clauses 11-15,further comprising the steps of generating a confidence value for eachsample of training data based on at least one activation level includedin the portion of activation data corresponding to the sample oftraining data, receiving a selection of a first confidence criterion viathe graphical user interface, determining a subset of samples oftraining data having confidence values that meet the first confidencecriterion, and updating the graphical user interface to remove allsamples of training data that are not included in the subset of samplesof training data.

17. The non-transitory computer-readable medium of any of clauses 11-16,further comprising the steps of generating a confidence value for eachsample of training data based on a comparison between a first activationlevel included in the portion of activation data corresponding to thesample of training data and a plurality of other activation levelsincluded in the portion of activation data corresponding to the sampleof training data, and updating the graphical user interface to displayeach sample of training data based on the confidence value generated forthe sample of training data.

18. The non-transitory computer-readable medium of any of clauses 11-17,further comprising the steps of receiving a logical expression via thegraphical user interface, evaluating the logical expression to generatean expression value for each sample of training data based on theportion of activation data corresponding to the sample of training data,and updating the graphical user interface to display each sample of thetraining data based on the expression value generated for the sample oftraining data.

19. The non-transitory computer-readable medium of any of clauses 11-18,wherein the step of updating the graphical user interface to displayeach sample of the training data comprises ranking each sample oftraining data based on the expression value generated for the sample toproduce a plurality of ranked samples of training data, and displayingthe plurality of ranked samples of training data.

20. Some embodiments include a system, comprising a memory storing asoftware application, and a processor that, when executing the softwareapplication, is configured to perform the steps of causing a neuralnetwork to execute an inference operation based on a plurality ofsamples of training data to generate a plurality of portions ofactivation data, wherein each sample of training data corresponds to adifferent portion of activation data, generating a position value foreach sample of training data based on the portion of activation datacorresponding to the sample of training data, and generating a graphicaluser interface that displays each sample of training data positionedwithin the graphical user interface based on the position valuegenerated for the sample of training data.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present invention andprotection.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “module,” a“system,” or a “computer.” Furthermore, aspects of the presentdisclosure may take the form of a computer program product embodied inone or more computer readable medium(s) having computer readable programcode embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine. The instructions, when executed via the processor ofthe computer or other programmable data processing apparatus, enable theimplementation of the functions/acts specified in the flowchart and/orblock diagram block or blocks. Such processors may be, withoutlimitation, general purpose processors, special-purpose processors,application-specific processors, or field-programmable gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A computer-implemented method for analyzing how aneural network has been trained, the method comprising: causing theneural network to execute an inference operation based on a plurality ofsamples of training data to generate a plurality of portions ofactivation data, wherein each portion of activation data includes a setof activation levels associated with possible outputs of the neuralnetwork, wherein at least a portion of the training data was previouslyused to train the neural network, and wherein each sample of thetraining data corresponds to a different portion of activation data;generating a position value for each sample of the training data basedon the portion of activation data corresponding to the sample of thetraining data; and generating a graphical user interface that displayseach sample of the training data positioned within the graphical userinterface based on the position value generated for the sample of thetraining data.
 2. The computer-implemented method of claim 1, whereingenerating the position value for each sample of the training datacomprises performing a dimensionality reduction operation based on theplurality of portions of activation data.
 3. The computer-implementedmethod of claim 1, wherein a given portion of activation data comprisesan N-dimensional vector, wherein N is a positive integer greater thantwo, and wherein the graphical user interface includes a two-dimensionalprojection of the plurality of portions of activation data.
 4. Thecomputer-implemented method of claim 1, wherein generating the graphicaluser interface comprises generating a t-distributed stochastic neighborembedding (t-SNE) map based on the plurality of portions of activationdata.
 5. The computer-implemented method of claim 1, wherein generatingthe graphical user interface comprises: ranking each sample of thetraining data based on the position value generated for the sample ofthe training data to produce a plurality of ranked samples of thetraining data; and positioning each ranked sample of the training datawithin a grid.
 6. The computer-implemented method of claim 1, furthercomprising: generating a confidence value for each sample of thetraining data based on at least one activation level included in theportion of activation data corresponding to the sample of the trainingdata; receiving a selection of a first confidence criterion via thegraphical user interface; determining a subset of samples of thetraining data having confidence values that meet the first confidencecriterion; and updating the graphical user interface to remove allsamples of the training data that are not included in the subset ofsamples of the training data.
 7. The computer-implemented method ofclaim 6, wherein determining the subset of samples of the training datacomprises identifying at least one sample of the training data having aconfidence value that is greater than a confidence thresholdcorresponding to the confidence criterion.
 8. The computer-implementedmethod of claim 6, wherein determining the subset of samples of thetraining data comprises identifying at least one sample of the trainingdata having a confidence value that is less than a confidence thresholdcorresponding to the confidence criterion.
 9. The computer-implementedmethod of claim 6, wherein determining the subset of samples of thetraining data comprises identifying at least one sample of the trainingdata corresponding to a portion of activation data that indicates anincorrect output of the neural network when the inference operation isexecuted.
 10. The computer-implemented method of claim 1, furthercomprising: generating a confidence value for each sample of thetraining data by determining a difference value between a firstactivation level included in the portion of activation datacorresponding to the sample of the training data and a second activationlevel included in the portion of activation data corresponding to thesample of the training data; and updating the graphical user interfaceto display each sample of the training data based on the confidencevalue generated for the sample of the training data.
 11. Anon-transitory computer-readable medium storing program instructionsthat, when executed by a processor, cause the processor to analyze how aneural network has been trained by performing the steps of: causing theneural network to execute an inference operation based on a plurality ofsamples of training data to generate a plurality of portions ofactivation data, wherein each portion of activation data includes a setof activation levels associated with possible outputs of the neuralnetwork, wherein at least a portion of the training data was previouslyused to train the neural network, and wherein each sample of thetraining data corresponds to a different portion of activation data; andgenerating a graphical user interface that displays each sample of thetraining data positioned within the graphical user interface based on aposition value generated for each sample of the training data based onthe portion of activation data corresponding to the sample of thetraining data.
 12. The non-transitory computer-readable medium of claim11, wherein generating the position value for each sample of thetraining data comprises performing a dimensionality reduction operationbased on the plurality of portions of activation data.
 13. Thenon-transitory computer-readable medium of claim 11, wherein a givenportion of activation data comprises an N-dimensional vector, wherein Nis a positive integer greater than two, and wherein the graphical userinterface includes a two-dimensional projection of the plurality ofportions of activation data.
 14. The non-transitory computer-readablemedium of claim 11, wherein the step of generating the graphical userinterface comprises generating a t-distributed stochastic neighborembedding (t-SNE) map based on the plurality of portions of activationdata.
 15. The non-transitory computer-readable medium of claim 11,wherein the step of generating the graphical user interface comprises:ranking each sample of the training data based on the position valuegenerated for the sample of the training data to produce a plurality ofranked samples of the training data; and positioning each ranked sampleof the training data within a grid.
 16. The non-transitorycomputer-readable medium of claim 11, further comprising the steps of:generating a confidence value for each sample of the training data basedon at least one activation level included in the portion of activationdata corresponding to the sample of the training data; receiving aselection of a first confidence criterion via the graphical userinterface; determining a subset of samples of the training data havingconfidence values that meet the first confidence criterion; and updatingthe graphical user interface to remove all samples of the training datathat are not included in the subset of samples of the training data. 17.The non-transitory computer-readable medium of claim 11, furthercomprising the steps of: generating a confidence value for each sampleof the training data based on a comparison between a first activationlevel included in the portion of activation data corresponding to thesample of the training data and a plurality of other activation levelsincluded in the portion of activation data corresponding to the sampleof the training data; and updating the graphical user interface todisplay each sample of the training data based on the confidence valuegenerated for the sample of the training data.
 18. The non-transitorycomputer-readable medium of claim 11, further comprising the steps of:receiving a logical expression via the graphical user interface;evaluating the logical expression to generate an expression value foreach sample of the training data based on the portion of activation datacorresponding to the sample of the training data; and updating thegraphical user interface to display each sample of the training databased on the expression value generated for the sample of the trainingdata.
 19. The non-transitory computer-readable medium of claim 18,wherein the step of updating the graphical user interface to displayeach sample of the training data comprises: ranking each sample of thetraining data based on the expression value generated for the sample toproduce a plurality of ranked samples of the training data; anddisplaying the plurality of ranked samples of the training data.
 20. Asystem, comprising: a memory storing a software application; and aprocessor that, when executing the software application, is configuredto perform the steps of: causing a neural network to execute aninference operation based on a plurality of samples of training data togenerate a plurality of portions of activation data, wherein eachportion of activation data includes a set of activation levelsassociated with possible outputs of the neural network, wherein at leasta portion of the training data was previously used to train the neuralnetwork, and wherein each sample of training data corresponds to adifferent portion of activation data, generating a position value foreach sample of the training data based on the portion of activation datacorresponding to the sample of the training data, and generating agraphical user interface that displays each sample of the training datapositioned within the graphical user interface based on the positionvalue generated for the sample of the training data.